Field of the Invention
This invention relates to vision substitution systems, specifically to a method and apparatus for presenting audio and tactile representations of items such as visual items found within visual representations and non-visual items that are components of non-visual entities, for example for use by blind people.
Description of Related Art
The World Health Organization estimates that there are about 39 million blind people in the world. Devices have previously been invented that substitute for aspects of vision with another sense, particularly hearing and touch, and can be useful to blind and partially sighted people, and to those temporarily unable to see, for example aircraft pilots blinded by smoke. The approach is known as “sensory substitution” or “vision substitution”. U.S. Pat. No. 1,350,954 discloses a reading machine that presents the shapes of letters by scanning along lines of type with a column of five spots of light, the reflected light from each spot controlling the volume of a different musical note, producing characteristic sequences of notes for each letter. U.S. Pat. No. 5,097,326 discloses presenting live images via sound. U.S. Pat. No. 6,963,656 discloses using musical sequences to convey features of images. Similar approaches have been used to sonify the lines on a two-dimensional line graph. Typically a left-to-right column scan is performed, height is mapped to pitch, and brightness to volume (either dark- or light-sounding). Horizontal lines produce a constant pitch, vertical lines produce a short blast of many frequencies, and the pitch of the sounds representing a sloping line will change frequency at a rate that indicates the angle of slope.
In the tactile modality, several inventions have allowed visual features to be presented via touch, usually via an array of tactile actuators (typically vibro-tactile or electro-tactile) acting on parts of the user's body, for example Telesensory's™ Optacon™ finger-read vibro-tactile array; U.S. Pat. No. 6,430,450 and Wicab's™ Brainport™ tongue-placed electro-tactile display; U.S. Pat. Appl. No. 2010/0151426 and EyePlusPlus's™ Forehead Sensory Recognition System electro-tactile display; electromechanical tactile abdomen displays; and KGS's™ 8 by 8/64-dot tactile display. The present inventor's U.S. Pat. No. 8,239,032 describes a palm-read tactile display. Furthermore, standard programmable braille displays 45 FIG. 3 can be configured to act as displays of simple visual representations.
The present inventor's U.S. Pat. No. 8,239,032 entitled “Audiotactile Vision Substitution System” discloses a system for presenting one or several apparently-moving speech sounds and corresponding tactile effects 48 & 31 FIG. 3, referred to as “tracers” 32 & 35 and “polytracers” 33, that trace out the shapes of lineal features (for example outlines of items 34), or symbolic shapes 35, at the same time as presenting information 36 related to the items, with distinct audiotactile indicator effects highlighting corners 37 within the shapes. The present inventor's UK Pat. No. GB2477431 entitled “Improved Audiotactile Vision System” discloses using buzzing tracers and outputting distinct effects when tracers cross the borders 38 FIG. 3 between elements of a regular lattice, to improve the perception of shapes. The present inventor's U.S. Pat. No. 9,430,954 entitled “System for Presenting Visual Items” discloses presenting visual items via groups of stereophonic sounds 39 FIG. 3 and tactile effects 40, referred to as “imprints”, which can be speech-like sounds 41. The dispersed effect of the range of pitches and stereo locations of the voices resembles the sounds of a crowd of people speaking in unison, and rapidly conveys the properties, approximate size, shape, and location, of the presented items. “Layouts” 42 FIG. 3 present the locations of items' content, via plain or coded speech sounds 43 & 44, braille 45 (or via an “Optacon” or other tactile array), or coded Morse-like taps 46 & 47.
The specification and drawings of U.S. Pat. No. 8,239,032, UK Pat. No. GB2477431, and U.S. Pat. No. 8,239,032 are incorporated by reference, and copies are obtainable from the Internet and elsewhere.
The “GATE” (Graphics Accessible To Everyone) project uses a graphical structural hierarchy defined by ontology to inform the user about the graphical content in a picture in a non-visual way. The Web Ontology Language “OWL” provides a technical framework for annotating pictures with a semantic hierarchy that includes attributes of described objects. An approach used by the US Navy for attending to two or more voices is to accelerate each voice, and then serialize them.